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1 



Monitoring unit alongside an assistance system 



2 



for motor vehicles 



3 

4 The present invention relates to a monitoring unit for the 

5 outside in a motor vehicle's direction of travel, which unit 

6 includes at least one camera system having an image-recording 

7 sensor. 
8 

9 The present invention further relates to an assistance system 

10 for motor vehicles, in particular for traffic-sign and/or 

11 traffic-lane registering, having at least one monitoring unit 

12 registering the outside in the motor vehicle's direction of 

13 travel, with said monitoring unit including at least one cam- 

14 era system having an image-recording sensor. 



16 Intelligent Advanced Driver Assistance Systems (ADAS) will 

17 play an increasingly important role in up-to-date motor vehi- 

18 cles of the future. Said vehicles will contain, for example, 

19 monitoring units such as camera systems having, for instance, 

20 digital CMOS (Complementary Metal-Oxide Semiconductor) or CCD 

21 (Charge-Coupled Device) image sensors as aids that register 

22 the outside, for example in the motor vehicle's direction of 

23 travel. 



25~ Pure black/white (B/W) - that is to say monochrome - image re- 

26 cording is for most tasks more favorable than color image re- 

27 cording in association with image-processing systems of said 

28 type employed in the automobile sector for registering travel 

29 environments. However, applications do exist where color in- 

30 formation obtained from, for example, the three RGB primary 

31 colors red (R) , green (G) , and blue (B) , and/or other colors 

32 such as, for instance, yellow (Y) etc. can be important for 

33 attaining a higher confidence level for the output vector gen- 



15 



24 
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1 erated by an image-processing means. 
2 

3 An instance thereof is traffic-sign recognition, with its be- 

4 ing possible to recognize by way of the, where applicable, in- 

5 dividual color information (R and/or G and/or B, and/or other 

6 colors such as, for instance, Y etc.) whether the sign con- 

7 cerned is a prohibition sign or a sign giving orders, or one 

8 that purely provides information. 
9 

10 Another function is recognizing colored lane markings associ- 

11 ated with, for example, roadworks. Color information is help- 

12 ful here, too, and necessary to be able, for example, to dis- 

13 tinguish between the normal white markings that are no longer 

14 valid and the additional yellow lane markings. 
15 

16 Conversely, purely monochrome (B/W) image recording will suf- 

17 fice for the purpose of recognizing objects such as obstacles, 

18 other vehicles, and cyclists or other persons because color 

19 information (R, G, B, Y etc.) will as a rule not provide bet- 

20 ter recognition quality. 
21 

22 The object of the invention is to provide an improved monitor- 

23 ing unit for the outside in a motor vehicle' s direction of 

24 travel, which unit includes at least one camera system having 

25 an image-recording sensor, and further to provide an improved 

26 assistance system for motor vehicles, in particular for traf- 

27 fic-sign and/or traffic-lane registering. 
28 

29 Said object is achieved by means of the features of the inde- 

30 pendent claims. 
31 

32 Advantageous embodiments and developments that can be used 

33 singly or in combination with each other are the subject of 
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1 the dependent claims. 
2 

3 The invention builds on generic monitoring units for the out- 

4 side in a motor vehicle's direction of travel in that the 

5 monitoring unit includes at least one camera system having an 

6 image-recording sensor having color coding (R, G, B, Y, . ..) 

7 in partial areas but otherwise monochrome coding (B/W) . The 

8 present invention thus proceeds from an image-recording sensor 

9 that is substantially embodied as monochrome (B/W) or, as the 

10 case may be, coded and inventively has color coding (R and/or 

11 G and/or B and/or other colors such as, for instance, Y etc.) 

12 in partial areas. 
13 

14 For the purpose of recognizing or, as the case may be, assign- 

15 ing specific colors of traffic signs it is proposed providing 

16 a color coding (R and/or G and/or B and/or other colors such 

17 as, for instance, Y etc.) of vertical stripes and/or areas on 

18 the right-hand and/or left-hand image edge. Because traffic 

19 signs move from the center of the image outward from the per- 

20 spective of a camera mounted in the front area of a motor ve- 

21 hicle traveling in a straight line, the color of the sign and 

22 the basic information associated therewith (prohibition, or- 

23 ders, other information) can be advantageously determined when 

24 . the sign is located within the color-coded stripes or, as the 

25 case may be, areas. 
26 

27 For the purpose of recognizing a lane marking's for instance 

28 yellow or white color it is proposed providing a color coding 

29 (R and/or G and/or B and/or other colors such as, for in- 

30 stance, Y etc.) of horizontal stripes and/or areas on the sen- 

31 sor' s bottom image edge, preferably in the area where the cam- 

32 era has a view onto the road directly over the hood, particu- 

33 larly in the case of a customary automobile. The camera can, 
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1 of course, be arranged analogously when the inventive monitor- 

2 ing unit is employed in a truck or van etc. As said area is 

3 not absolutely essential for image evaluating in terms of ob- 

4 ject detecting or lane registering it can advantageously be 

5 used for color-recognition purposes. Because traffic lanes can 

6 be seen from the center/top of the central image area down to 

7 the bottom left-hand or, as the case may be, right-hand image 

8 area when the camera is mounted on the front of a vehicle 

9 traveling in a straight line, then in addition to the traffic 

10 lane's position the color can advantageously be determined and 

11 made available to an image-processing system. 
12 

13 In order to obtain the desired color information (R and/or G 

14 and/or B and/or other colors such as, for instance, Y etc.) it 

15 is proposed attaching a tiny color filter, where applicable 

16 specifically accommodated to the application, in front of each 

17 individual cell (pixel) of the stripes or, as the case may be, 

18 areas concerned. 
19 

20 The color-coded stripes and/or areas are in a first embodiment 

21 embodied as, for example, a single color (R; Y; etc. ...). 
22 

23 The vertical stripes and/or areas expediently have, for exam- 

24 pie, a red (R) color coding and the horizontal stripes and/or 

25 areas preferably a yellow (Y) color coding. 
26 

27 The color-coded horizontal and vertical stripes and/or areas 

28 can alternatively or additionally be embodied as a combination 

29 of two colors (R, G) . In particular vertical stripes and/or 

30 areas having red (R) , and green (G) color coding have proved 

31 useful for increasing the contrast of signs that give orders 

32 and are placed in front of trees. 
33 
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1 For obtaining the desired color information (R, G, B) it is 

2 proposed in a further embodiment to arrange a tiny color fil- 

3 ter in one of the three RGB primary colors red (R) , green (G) , 

4 and blue (B) in front of each individual cell (pixel) of the 

5 stripes or, as the case may be, areas concerned, with said 

6 filters being arranged preferably in what is termed the "Bayer 

7 pattern". 
8 

9. The invention further comprises an assistance system having a 

10 monitoring unit of the aforementioned type. The monitoring 

11 unit's advantages will in this way also come to bear within 

12 the scope of an overall system, in particular for traffic-sign 

13 and/or traffic-lane registering. The ratio of monochrome cod- 

14 ing (B/W) to partial color coding is therein preferably 

15 80:20%. Depending on the focus of the specific application the 

16 partially color-coded areas can also occupy 25% or up to 40% 

17 of the sensor surface. 
18 

19 The main advantage of an inventive monitoring unit for the 

20 outside in a motor vehicle' s direction of travel or, as the 

21 case may be, of an assistance system for motor vehicles in- 

22 eluding said type of monitoring unit having a partially color- 

23 coded (R and/or G and/or B and/or other colors such as, for 

24 instance, Y etc.) camera is that all relevant data for regis- 

25 tering travel environments can for the first time be obtained 

26 using just one camera. Owing to the camera's substantially 

27 monochrome (B/W) image recorder there will be no constraints 

28 on sensitivity so that reliable evaluating will be ensured 

29 even in poor light conditions; the color coding (R and/or G 

30 and/or B and/or other colors such as, for instance, Y etc.) in 

31 the sensor's edge area will not compromise applications for 

32 which the purely monochrome (B/W) image is more favorable. 
33 
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1 By contrast, color coding (R and/or G and/or B and/or other 

2 colors such as, for instance, Y etc.) having vertical stripes 

3 and/or areas on the right-hand and/or left-hand edge of the 

4 sensor's image field will provide reliable information about 

5 the color (R, G, B, Y, ...) of traffic signs; the color coding 

6 (R, G, B, Y, . . . ) in the bottom image area will provide reli- 

7 able information about the color of lane markings. 
8 

9 A single camera can thus be used for all relevant applica- 

10 tions, and that will advantageously save costs and mounting 

11 space. 
12 

13 The invention is described below with reference to the accom- 

14 panying drawing and with the aid of a preferred exemplary em- 

15 bodiment . 
16 

17 Fig. 1 is the exemplary image, shown in monochrome 

18 (B/W) , of a scene outside a motor vehicle; 
19 

20 Fig. 2 is a schematic sketch of a possible color-filter 

21 distribution (R, G, B) of the image recorder of 

22 a monitoring unit's camera; 
23 

24- Fig. 3 shows an enlarged section of the color-filter 

25 distribution according to Fig. 2; 

26 

27 Fig. 4 is the exemplary image according to Fig. 1 show- 

28 ing a scene outside that has been partially 

29 color-coded according to Fig. 2; and 
30 

31 Figs. 5 and 6 are further instances of a color-filter distri- 

32 bution of a monitoring unit's image recorder. 
33 
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1 Fig. 1 is the exemplary image, shown in monochrome, of a scene 

2 outside a motor vehicle. What is disadvantageous about 

3 black/white (B/W) image recording of said type is the lack of 

4 color information that could be obtained from, for example, 

5 the three primary colors red (R) and/or green (G) and/or blue 

6 (B) and/or other colors such as, for instance, Y etc. 
7 

8 Color information (R, G, B, Y, . . . ) of said type can be impor- 

9 tant for attaining a higher confidence level for the output 

10 vector generated by an image-processing means, in particular 

11 for recognizing traffic signs, with its being possible to rec- 

12 ognize by way of the color information (R and/or G and/or B 

13 and/or other colors such as, for instance, Y etc.) whether the 

14 sign concerned is a prohibition sign or a sign giving orders - 

15 as in the case of the speed-limit sign on the right of the im- 

16 age section shown Fig. 1 - or one that purely provides infor- 

17 mation (not shown) . 
18 

19 A further function is recognizing colored lane markings asso- 

20 ciated with roadworks. It is helpful here and necessary to 

21 distinguish between the normal white markings that are no 

22 longer valid and the additional yellow lanes. 
23 

24 . Conceivable solutions featuring an exclusively color-coded (R, 

25 G, B, Y, . . . ) image recorder are not only more compute-bound 

26 and hence more expensive; they also have the disadvantage that 

27 monochrome (B/W) images are more favorable for a number of 

28 outside applications, in particular for the purpose of detect- 

29 ing objects such as obstacles, other vehicles, and cyclists or 

30 other persons and the like, or night-time applications. 
31 

32 To resolve this conflict of requirements the present invention 

33 proposes the use in a generic monitoring unit of a, for exam- 
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1 pie, specially embodied CCD sensor 10 having color coding (R, 

2 G, B, Y, . . . ) in partial areas 11, 12, 13 but otherwise mono- 

3 chrome coding (B/W) . The present invention in other words pro- 

4 ceeds from a sensor 10 that is embodied as substantially mono- 

5 chrome (B/W) or, as the case may be, coded and has color cod- 

6 ing (R, G, B, Y, . ..) in partial areas. That could be achieved 

7 by means of, for instance, a color coding (R, G, B, Y, . . .) of 

8 vertical stripes (not shown) or areas on the right-hand 11 and 

9 left-hand 12 image edge, and would be helpful, for example, 

10 for assigning traffic signs the correct color (R, G, B, Y, 

11 . ..). Because the traffic signs move from the center of the 

12 image outward from the perspective of a camera mounted in the 

13 front area of a motor vehicle traveling in a straight line, 

14 the color (R, G, B, Y, . ..) of the sign can be determined when 

15 the sign is located within the color-coded (R, G, B, Y, . ... ) 

16 stripes or, as the case may be, areas 11, 12. 
17 

18 The expedient approach for recognizing the color of the traf- 

19 fic lane (yellow or white, for example) is also to provide 

20 color coding, in particular yellow (Y) coding, in the bottom 

21 area 13 of the sensor 10, preferably in the area where the 

22 camera has a view onto the road directly over the hood in the 

23 case of a customary automobile. As said area is not absolutely 

24 essential for image evaluating in terms of object detecting or 

25 lane registering it can be used for color-recognition (Y) pur- 

26 poses. Because traffic lanes can be seen from the center /top 

27 of the central image area down to the bottom left-hand and 

28 right-hand image area when the camera is mounted on the front 

29 of a vehicle traveling in a straight line, then in addition to 

30 the traffic lane's position the color can be determined and 

31 made available to an image-processing system. This feature is 

32 not, of course, restricted to motor vehicles having a hood but 

33 can be realized analogously when the inventive monitoring unit 



PCT/EP2005/052080 / 2004 P0 987 1W0US 

9 

1 is employed in particular in a truck or van etc. 
2 

3 Especially in the case also of night-vision applications the 

4 best possible sensitivity is necessary across the entire, 

5 which is to say unfiltered, wavelength spectrum, including the 

6 near infrared. Since, though, a much smaller angular range 

7 generally has to be detected here, it suffices to provide a 

8 smaller central area with no color coding. 
9 

10 Fig. 2 is a schematic sketch of a possible color-filter dis- 

11 tribution (R, G, B) of the image recorder 10 of a monitoring 

12 unit's camera. The light-sensitive cells of a non color-coded 

13 (N) CCD sensor 10 in the monitoring unit's digital camera nor- 

14 mally only register brightness values. In order to obtain 

15 color information (R, G, B, Y, ...) a tiny color filter is to 

16 be attached in front of each individual cell for example in 

17 one of the three RGB primary colors red (R) , green (G) , and/or 

18 blue (B) , and/or another color such as, for instance, yellow 

19 (Y) etc. Filters constructed from primary colors are usually 

20 attached in the arrangement R-G-R-G and, in the next cell, G- 

21 B-G-B, and in this arrangement produce what is termed the 

22 "Bayer pattern". 
23 

24 Fig. 3 shows an enlarged section of the color-filter distribu- 

25 tion (R, G, B) according to Fig. 2 arranged in the Bayer pat- 

26 tern. It can clearly be seen how each CCD element only sup- 

27 plies the information for a single color component so that the 

28 adjacent pixels accordingly have to be used for calculating 

29 the actual color. This process is referred to as "color inter- 

30 polating". 
31 

32 Color information useful in the context of road traffic can be 

33 obtained within the Bayer-pattern coded stripes or, as the 
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1 case may be, areas 11, 12, 13 of the sensor 10 by means of 

2 color interpolating of said type and made available to an im- 

3 age-processing system: 
4 

5 The corresponding exemplary image according to Fig . 1 showing 

6 a scene outside that has been partially color-coded according 

7 to Fig. 2 is shown in Fig. 4. 
8 

9 The color information does not have to consist of the three 

10 primary colors: It is also possible only to use single color 

11 filters such as, for instance, red (R) for signs giving orders 

12 or yellow (Y) for lane markings associated with, for example, 

13 roadworks . Combinations of red (R) and green (G) color filters 

14 or others specifically accommodated to the application have 

15 also proved useful for increasing the contrast of signs that 

16 give orders and are placed in front of trees . Color filters of 

17 said type can advantageously also be arranged spaced apart, 

18 being located, for example, on every other pixel in a row 

19 and/or column; that means in any combination with non-coded 

20 (N) pixels or pixels coded in another color or, as the case 

21 may be, where applicable specifically accommodated color fil- 

22 ters. 
23 

24 Arrangements and further instances of color coding that offer 

25 this type of advantage and can be used instead of the "Bayer 

26 pattern" are shown in Figs. 5 and 6: 
27 

28 Fig . 5 shows clearly how only every other pixel in a row 

29 and/or every other column has been provided with red color 

30 filters (R) for recognizing, for instance, signs giving or- 

31 ders. 
32 

33 Fig. 6 shows the advantageous embodiment of a combination of 
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1 two colors. Instead of R-G-R-G and in the next row G-B-G-B as 

2 in the case of the "Bayer pattern", color filters are used in 

3 the pattern R-N-R-N and in the next row N-G-N-G, with N signi- 

4 fying no color coding. 
5 

6 The main advantage of a partially color-coded camera is that 

7 all relevant data for registering travel environments can be 

8 obtained using just one camera. Owing to the camera's substan- 

9 tially monochrome (B/W) image recorder there is will be no 

10 constraints on sensitivity so that reliable evaluating will be 

11 ensured even in poor light conditions; the color coding (R, G, 

12 B, Y, . . . ) in defined edge areas 11, 12, 13 of the sensor 10 

13 will not compromise applications for which the purely mono- 

14 chrome (B/W) image is more favorable. By contrast, color cod- 

15 ing (R, G, B, Y, . . . ) having vertical stripes on the left-hand 

16 11 and right-hand 12 edge of the sensor' s image field will 

17 provide reliable information about the color (R, G, B, Y, . ..) 

18 of traffic signs; the color coding (R, G, B, Y, ...) in the 

19 bottom image area 13 will provide reliable information about 

20 the color of traffic lanes. 
21 

22 A single camera can thus be used for all applications, and 

23 that will advantageously save costs and mounting space. 
24 

25 The present invention is thus especially suitable for imple- 

26 mentation in an assistance system for motor vehicles, in par- 

27 ticular for traffic-sign and/or traffic-lane registering. It 

28 will advantageously increase road-traffic safety not only in 

29 combination with existing assistance systems for motor vehi- 

30 cles such as blind-spot detection, LDW (Lane Departure Warn- 

31 ing) lane monitoring, night vision etc. 



